{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4ef03480-a7f5-401f-bd7f-563a2ed4eb2f",
   "metadata": {},
   "source": [
    "# 语言模型的目标\n",
    "\n",
    "语言模型的目标是建模自然语言的概率分布"
   ]
  },
  {
   "attachments": {
    "ad44b9b3-fffd-465e-813e-1fa785bafb31.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAARMAAAD3CAYAAAA+C7CYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAOxAAADsQBlSsOGwAAIABJREFUeJzs3Xd4VMXewPHv9t30HhISIJCQ0Htv0gVEqnSlWFERvPjq\nFcXeALsIoqKCgIB4EREbIFVCkxJaQhJSgPTetu++f6wEIiGNTTbAfJ6Hh7Nn58yZs8n+MmdmzozE\narVaEQRBuElSRxdAEITbgwgmgiDYhQgmgiDYhdzRBRBuHzExMUydOpXBgwdTUFBAeno6qampbNmy\nheeee46PPvqII0eOcPbsWZ544gkmTpyIVCqlWbNmBAUFERMTQ1BQEOnp6cyZM4f169ezYcMGunXr\nRqNGjbh06RIPPfQQ7dq1c/SlCuUQNRPBbsLCwggJCSEiIqL09ciRI9m1axcAcXFxXLx4kYyMDGJj\nYzEYDGRnZxMUFMSePXvIyMggJyeHgIAAgoKC0Gg06PV6srOzadu2LQUFBYSFhTnyEoUKiJqJYDdR\nUVGkp6fzxx9/YDabSU9Pp1u3brRo0QI3Nzd27NhBfHw8aWlpbNmyhYiICFatWoWTkxMymQy5XM6J\nEydQKBT069ePU6dO4efnh4+PD3v37sXLywuDwYCTk5OjL1Uoh+yVV155xdGFEG4PXl5ehIeHk5OT\nQ3Z2NhaLhQYNGhAeHs7q1at56623cHZ2xtXVlXnz5rF582bCwsKYPn06Go2G7t27k5WVxaBBg+jQ\noQMNGjRAJpORm5tLREQEvXv3Jjc3l+DgYEdfqlAOUTMR7CYmJobPP/8crVaL0WhEIpGg0WiwWq3k\n5uZelz4nJ4euXbuSkZHBt99+i7OzM926dSM/P5+lS5fi7OxMVlYWycnJNG7cmAMHDtCnTx8HXJlQ\nFRIxaE2wp7i4OPbs2cPx48fR6/XodDqWLVvG4sWLycjIwGQyYbFYUCqVHDt2jGnTpjFo0CAMBgNf\nffUV77//PrNnz2bmzJn06tWLs2fP8uuvvzJ//nxHX5pQCRFMBLuyWq1YrVYkEgkSicRuedorL6H2\niGAiCIJdiK5hwWEsFgs5OTmOLoZgJyKYCA6zYcMG3n//fUcXQ7ATEUwEh7BYLJw8eZKCggJRO7lN\niDYTwSGys7NxdXUlPz8fk8lEQECAo4sk3CQRTASHyszMxNfX19HFEOxA3OYIgmAXYgSsUC3FuiJM\nZqPdxn0UafNRlSjskheAm5OH3fISqkcEE6HKYi+fYcFXM3F1csdisTi6ONcp1hXy5L0v06/dCEcX\n5Y4kgolQZQu+msnMEfNo06yzo4tSrrzCbF77ei6dw/virHZ1dHHuOKLNRKgybzc/Gvo2rlLaQ38d\n4dXn37puf15ePo8+MIdLyZfJy8u3a/k8XL1RKlRIpTK75itUjQgmgt2ZzWaWvr+C+QueAiAjPZOz\np6IB2LRuM02aNiY2Jo7XX3iHi0mXOH70JAaDgZPHT2E0GtHp9Bz66wipl9PIy80jKzOb7CwxFqW+\nE7c5gt3JZDJCmjVmyRsfMnbivaz9ZiOhzZvy195ItCVaJEgoLi5Br9Mz95H/Y+Ddd7H49Q8YOPQu\n9u06QJduHbkQl0BgUADvvPoeBr2BV9550dGXJVRC1EyEWvHa4oWMGH03kfsOM3XGBB6cPZ2o46dp\nHhFGu45taN22Jc3CQujaozOTH7iP0OZNmTZzEkkJycTFXiAxIZngxkFoS3QEBgXi5i7aQOo7MWhN\nqLJHPxzBk+NfxMPVu9K0+XkFWK0WPDw9MBqNGA1GnJydMBqNACgUCvLzClCplKjUKnRaHRonDdoS\nLWqNmqLCYhQKOQqlArPZDIBSqaz0vP/5eBqrnv0TJ5XLzV2sUG3iNkeoskCfJiz731uMvWs6FmvV\nuoZTrm1jzaydcl0RnRQFgFKurt0TCeUSNROhygwmPfOW3YeXmz8qRf37wsqkMh4fuRAPl8prToL9\niWAiOExqaioJCQn07NnT0UUR7EA0wAoO88EHH7BkyRJHF0OwE1EzERwiOzubo0ePAtC4cePShbuE\nW5cIJoJDWCwWpFIpmZmZeHt7I5WKSvKtTvwEBYe4NniIQHJ7ED9FQRDsQowzEaol/lIOBUV6u81n\nkpeXh0eO2S55yWVSWof62SUvofpEMBGq7Oe9Mby4dBctm/lhsdirqc0K2CcwRSdk8sDIdvzn/h52\nyU+oHtEAK1RZ+wmf8c7Tw2gWXD8HhekNJqb9dz17v56Jm7PK0cW544g2E6HK/L1dcK3ml3Trj9+X\nbq/49D17F6kMlVKOWilHLhO/1o4gPnWhVu3a8QtXKr8Dh4zAoNdz6WISp6OOYzAYuHwpmeSkBADS\nUlM4dvQgFouFSxeTyM3NdmTRhWoSbSZCnVn+yRL+8+zLzJ/zIKPHT+aXrT9gNpvRlpQw4+EnOBS5\njzNRJ0iIjyUp8QJTHnjY0UUWqkHUTIRa98Hi11jzzeelr7v16MPkaQ+Snp5KZmY6Xt4+gITft/2I\nm7s7F5MTAWgQEOiYAgs1IhpghSob8ti3vPrEEHw8nat8TElJMUaDAZlMhkwuR63WoNfpUGs0lJQU\nI5PKMFvMODk5U1xchEqlxmwyYbVaUWs01S7jff9Zw/5vZuHiVPncJ4J9idscoco6tghg9uubmTm6\nM5Z6+Dfo4MkkAJw19luHR6g6UTMRqsxisfLOV/sxW6yoFPVvBnipVMITE7ugUYtg4ggimAgOc/78\neWJiYhg5cqSjiyLYgWiAFRxm6dKlrFixwtHFEOxE1EwEh8jIyODy5cu2hla1mpYtWzq6SMJNEsFE\ncKjMzEx8fX0dXQzBDkRvjlCpgoICTp06RWpqKlqt1u55u7m52TXPqggODqZ169b4+PjU+blvV6Jm\nIlTqzTffJDw8nKCgINRqNfb8lcnNzcXT09Nu+VVVcnIyUVFRdO3alaFDh9b5+W9HIpgIFYqMjKRR\no0a1VnvIyspyWO1Ar9ezatUqQkNDGTVqlEPKcDsRvTnCDaWmprJjxw6H3IbUBZVKxaxZs7hw4QLn\nzp1zdHFueaLNRLihI0eO0L1799LXJl02cdvHYdLZb2k+q9VKmj1mbbNa0BddpuWYw2g8qz7TvVwu\np1evXhw+fJgWLVrcfDnuYCKYCDeUnJzM3XffXfr65LoQGoT0pHGXR7FWcXnQupSfGcfZzV1pf38q\nMkXVnx8KCQlh+/btpTPmCzUjPjnhhrRaLZprHrZTOgfQuOUINC6+OLn6l/57YuGvpdtxF438uvcy\n//sjASdX/zLbTq7+fPzNcXYfzuDBZ3/ifJKhdP/Q+1fzn9e34+Tqz0PPbcXJ1Z+TMUWonX2RKb3I\nyJNxMd3KxXQr8ZdMGCyumCXuZcoR0LQXUpkasLJx40ZycnKqdJ1qtRq9Xm/XhuU7kaiZCDfl0LEE\nTCYLefklrN9yhLx8LadjLnN3/1Z8tnoP+w/HcXf/VqXpAxu489hza1n84jg2/3qc9q2DAQgO9CSw\ngQc799naLgqLdMQnZbJqYyTPPzWMybO/5O0FY5j30kaemT2YiY9+Tse2jViz9MHryrRixQpOnjpP\nSkrKDcs9b948O38SgqiZCDdl1cZIMnMKWbf5MHsiz/PM7MGMGdYBgD2R50u3AQwGE1t+O0nSkbc5\nceYix04ll8nrv0/ezSdf7UKrM3LkRCLTxnUnN78EgN5dQ7mrZzgtmwcwbVx32rUKusGk1lby8vLQ\narVkZ2eX++/IkSO19nncyUTNRKgxrdZASnoeW1c9yb3TP2XiqM489eIGCot0DO3fkqF3teLHX08w\ntL9tqLxCIUOhkLFw0RZy8oqR/WuuVqVSzvNzhjHygaV8t+whFi7aQpf2TUrfA9uTwcB1x14l4bnn\nnmPNuh8YOXIkXl5e16XYtm2bfT4AoQwxzkS4oSVLljBlypTSruFTGyLoeNcc1M62L6jFYkGnM+Hk\npKSoWIeLs5rCIh0atQKLxYpSKcdkMpduX5GbV4ynhzNGoxnFP1MZFJfocXayTVZdUKjFzVVDfoEW\ndzcNVqsVnc6IRqMsTVdSYsCKtfSYK/b+8BRtpyQgU7jc8Lq2bdvGiBEjrrvWl19+GZms/k2tcKsQ\nNROhyuQqD1IT9hPcfDBWbL05CjkYDUZUCjAailErwWoxIwGMBj1AmW0AFydbWgCjwbZPKQejwQSA\nRmV730l9NZ1cZjvPlXSKf35zrxwDUJCdgMV89TxC3RLBRKiyZoN/4NSGFiRH/4FUVv/WpZHKNUTc\n82eFtRKh9ohgIlSZ0rkhnWYV2DVP8dTw7UMEE6HKDp9OZf4Hf9KhXYi9VvT8Z3XQ2JvORiqR8Pfx\nCzw5qRMTB1d9BKxgPyKYCFU2/4M/uffe7jRp4u/oopSrQ6fmfLV2J/f2DUWjEr/adU2MMxGqzNvD\nCS8vV0cX44acnFSYTBYxktVBRPgWasWWr96l590TOLj9f4ycPg+T0cCPK5dgtVjwbhBEQW4WcoWS\n5u26YTIaAdi37TsUShWjZs5n66oPuWv0A6QnxwPQse8wFCq1Iy9JqIQIJoLdGfRazp88hJOLG7qS\nIr77+GV6DBlLZkoyTi5uXI7cibd/QywWC7+fO4Gbpw8KlRqNkwv+wU05eWA7RoOOQ9v/R9dBYzi2\n5xciOvbCXQSTek3c5gh2F/n7D8gVClw8vLlv9osU5Wfj4dMA38BGmIwGigpyMeh1+AQE06xVR5zd\nPPH0DUChVGPU63Hx8MbVwxtdSTH+QSG4eXo7+pKEKhDBRLC7iI69aBLRnrY9BpKScJ6QiPZ4+Phj\n0GkJDAmnU78RBIaEYzToadN9IEa9jrSkOPJzMog+cQDfwEZYreDmKeZnvZWI2xzB7vyDQnBx82DX\n/1ahLSlEqVIT+fsmFCo1aqerA8qK8rLx9G2Af3BTigvz8PZvSEjLDpw9so/hU5/g1KFd5OdkoHFx\nR64UawfXdyKYCLVi8IRHqpx2wNgZZV4HNAoFoNewCUgkktLXQv0mgolgd7qSYkoK89j85WJMRgMT\nn3wFpVrDni3f0r7PUM4c3kP8mb9p3q47GZcScHK5OsesXqdl/GML+O7jl8m4dIEuA+5FqdLQuf89\nDrwioSpEMBHsLu1iPFEHduAb2BgPH3+2f/8Ffg2bcDkhhhN//UGbbgMw6LScPxGJVCqj17AJZKYk\n4eXfkD82rODs0b04ubiSn5OJxWIhNuqwCCa3ANEAK9idu5cvly6cQ+3kQnbaJaRSGT4BwTi7etBt\n0BgUKjWevgE0Dm+Lp18gX7/9NEUFufz09ft4+TXE0zeAC2eP0fmuEfy69lMCQ5pj0Nt38S/B/kQw\nEepEqy79cPfxw9u/IbqSQkxGAxdjzyCVSbFYLLi4e1FSlI9EKiEvOwOrFeLPHKP7kLFEHdhBfrb9\nZsQXaocIJkKtUChUqJ2cCWgShsViBiApJorzJw8SGBKOtriApq06cuHMMZ5avJroYwcY+/BzpCXH\nE9AolJ53jye8fQ/ueWAugSHh+AY2cvAVCZURbSZCtUiqsMaNp28AD7+0tPR198FjAZj92uel+7r0\nHwnAwHGzAJj81KsAhLXtVuYYgHGPPl/l8kmlEsSjOY4haiZClXm6qkhKynB0MW4oMzMfnd5UOk+s\nULdEzUSospce6cUDC38mNiYZpaJ+/epIpRJiL6Tz1pN9xfQDDiI+daHKQgLd2fbReKITc7BSv+4l\nrFYrQX7dCPKvv1Mk3O5EMBGqxc1FRdfWAXbJKzExkfj4eAYOHGiX/ATHEsFEcJjly5eTkJAggslt\nQgQTwSGysrIYO9bWYxMdHU1EhJi39VYngongED4+Pvj4+JCVlYWPj5hq4HYggkk9kpOTwy+//OKQ\nc0+dOrVKY0jsbe7cuahUdbsGT0pKynUr+gk3TwSTembatGkOOa+jJmEeOHAgrq6iB+Z2IIKJUC2G\n4hQsxiK7rZszbEBrPD097JKXRCJH5dbULnkJ1SeCiVBl+Rf/IG77eDRuQfaryVit5Njp9kpXeJmg\nLq/j32auXfITqkcEE6HK4raPp33/+Xj4Nnd0UcplMmrZv3ke3s0fQK7ydHRx7jgimFwjJyeHqKgo\nLl++jFZb9/NnFBcX4+zsbNc8VSoVfn5+tGnThsDAwJvKS+kcgMrJ67r9x08l06HN1ad6I4/Gk1eg\nZdiA1mW2AVLS8vhs9V76dAtlcL+Wpce8u/wPHpzcC08PZ77bfJjJY7pWu3xyhQapTI1EqgBAp9Oh\nVovlMeqKCCb/2L17NwcOHKBVq1ZERESg0WjqvFEyPz8fd3d3u+ap1+vJyMjg66+/5oUXXrBr3ld8\nvPJPvv5wBgBRZy+Rkp6PVmcAKLMNsPr7g3TvFMJX6//C19uV9q2DAdjy+0niEjL4bPE0tm6PYvKY\nrkQejadbxxCMRjOpGfno9SYADEYzwYGeyGRSXF3KDxZLlizhiSeeqJXrFcp3xweTixcv8s033zB7\n9mw6derk0LLk5OTg5XX9X/6bFRYWRq9evVi+fDndunWjY8eOdsv78f+uIzUjn2Xf7GbfoVi+/WQW\nsQm2J4snz/6C8fdc/UwNBhNHTiTy3JND2XswloWLt7B19ZMABAd68vEbkxg9cxkSiYQ/90czoHcE\nEx/9nMULx/Hp17tZ8tJ4pjz+JeuWPcTk2V8gkUhYt+yh68p07O9jZGdnc+LEiXLLfPDgQf7zn//Y\n7TMQbO74YHL48GH69OlT52MdHKF3797s37+fDh062GVMiVZrICU9j62rnuTe6Z8ycVRnnnpxA4VF\nOob2b8nQu1rx468nGNrfdjujUMhQKGQsXLSFnLxiZLKyM2AolXKenzOMkQ8s5btlD7Fw0Ra6tG9S\n+h5QOr3Av4+9VsdOHVGo3WjUqFG5wTk3N/emr1243h0fTJKSkujcuXONjtXlx3Hhz6lYzTq7lMVq\ntZJqp54Ni1mHoTiVdlMSkKttK+I1btyYH374AbPZjFxegx+91YJEcvVLrFLJWffpQygUMr7/4hFc\nnNWMG9ERjVqBxWJFqZQzbVw3LBbb7aJEImH9Zw+Tm1eMp4czRqO5NK8v3r0fgG4dQ4iLfAO1WsEz\ns4fg7ma73Xxx7nAAViy2jcP5fPH95T65bDHrwGqhffv21b8+4abc8cGkpKSkRo2eFpOWMz90JKzD\nJDz8wmuhZDdHIpGSdO5X4n4fRcSo/f/sk6DRaCgpKcHNza2SHK6n8W7HwZ8X0KLbTKxWS+n+gn/+\nL/pnmtbiKuSVlnf9vsJrtkuybf9rc67uyy8n3bXbWZdPAiBVuCDUvTs+mMDNjP6U0DCs/3V7M7IK\n8PO5+mX96ru/GNA7nCbBtmdQEi9mlW6fOH2R73/+m8mju9A6oiFga1tY9OnvvDhvOBKJpMa9G03b\njuXojiU1ubByNRu4ntg/xpKaloZUrrFLnka9AYXKPqv1STQRtJu6pUztSag7IphUU05ODrt27WLM\nqBFIpNd/fLl5xaz+/iDPzB6C1Wrl6MkkLqXmotOZSl+7Ol/tgfj0m93MmNCD1d8fpG/3MO4Z3BaD\n0cTna/bi5eHEEzP7l/ZupGXk08DPneycIgxGM/mFWtQqOemZhXRoHYxEIkGhkJXmfW3twR4kUjnN\n7/7JrnlmZmbi6+tr1zwFxxDB5F/MZjPLly+vMM25c+fIykyhSzm9kt/9eIQ9ked5ZvYQ5i7cQKe2\njflt1xnuG9mp9HXXDk1K0/t4uTDliZX8vm4uCxdv4Z7BbQEYPrANB45eYNRQ273/kROJfL5mHy2b\nB9A4yIuNP/3Ng5N78er7P3PPoLa88M6PDOwdwfNPDbPbZ/FvFqOOpGUPUhL/N1aLufIDqijdDnlY\nTbbu5xbvHkfuYv8eMaFyIpj8i9Fo5NVXXyUzs/x1WnJzc9m8eTMzHpjCiTWvlHnv1LnLXEjK4v1X\n7mP2c2txc1UTl5CBSilHggSNWkFcQgbdOoQAkJNbzKlzl5kypisfr/wTk7lsTWLtpw+y4tu9ZOUU\n4eKsorhET9NGPpgtFibc24nmzfzp16M5D0/tTWpGPvFJtbu2TPSzXWj99NeoPBvU6nluxtEXehGx\n6AgytWg3qWsimFSTp6cns2bNwmLWX9e9Gt7Mn7eeH41SKWfJS+NwcVZTWKTjhbnDUSplLHpxHIVF\nOpydbG0EXp7ObF39BAWFOtzdNBSX6AFwcVbzwSsTAHhkWh+mju2Ki7OaVR/NRKGQlfaCyOVSXpw7\nHLVawdvPj7murBIAq/1qEFK1Cze7jsSpc+cp0Wrp1rEdAOmZWfh6e3HkxCnatghHo7nJEatWK3Z7\nClGoFhFMashqMWIxG8hJP4ebl62mcWWFBZPRhFoJJqMOjQrAjMVsxmIGjQosZgPX3iU4aySYjDpU\nCtsxAMprtq/kJQFMRmPpV8VsAoUczCY9SsWVc9uOkUgkpCcdwqi7pjvEwZZ/8x27/jpI64gwPljx\nDcGBDZg05h6ef/M9XJ2daf5/TW4+mAgOI4JJDckULoQP/52oX4YikamQ1Le/hhIpcpUnrSecdnRJ\nAEjLyCTA35eQRkFkZufQtUNbnn50Bg/Pf5HHZ05l09bfUNmpV0dwDBFMboJLgx50mlVQecIqqq3h\n9PVBanomeyOPEJuQRNLFyyjkCmY/+zIjBt3Frzv3cj4+kd1/HUYmk9KpXWt8vMRTv7caEUxuglZv\n4te/LmAwmuySn8FgQKlMs0teaqWce/qGIq9g2HldatOiOYXFxfTv1Y2QRkGcOH2OAb274+/ng8Vi\nYdnXa5k6biSNggJR1GR0ruBw4qdWQyaThSGPbyCksS+e3tUfTXpj9hmafz7mMu+tOcKO5RNRKR3/\nYzYYjQzq05NT52KIibvAgnmP0TioIc+/8R5FJcXs3bKWles2ceTEKV6e/6SjiyvUgON/y25R6347\nS+fOYfTs2bLyxA7Qu3dr9u87zeZdsUwa2sJh5bicmk7DAH9OR8cyqG9PJo4eXvqe1Wrl7Rfnl75+\n58VnOHs+jqycXIIb2mehL6HuiGBSB7JSL7L/l/WMfvD/WLX4/5j+7BJ2b/kWo16LTluCrqQIZzcP\n2vccQl62bQjXoR0/YjYZGDzhEfb+tIYB42ah0jgTf+oIDZu1IKjprbHOzEdfrKJZk0acjj5P7IUk\n+nTvzP3j7+XLtZs4ExNHanoG0yeOYdr4e0m8eJlfdu6hYQN/2rWKoFV4mKOLL1SDCCZ1YMemL/H2\nb8imz97CydWdwzu3kJV6EaNex+XEGHwaBFNckEdhfg6nDu5EV1JMcUEebbr3R1dcSGF+Dps+e5MB\nY2YQGNKcY3t+uWWCSce2rRg3YgixCUns2HuAR++fRNTZaC4kXSS/sBAnJw1nY+I4ez6OM9FxHD1x\nGtqDwWAUweQWUz9a525jupJiDm3fTIvOfTHotdw3+0WCQ1sS2KQ5Rfk56EuKMRkNmIwGPLz98PYP\nwt3bj4597+bY3l+RK9WEtu6CxWympDAf/+Bba/b1Rg0D2fzLdpZ9vY7f/tzHu8tX0qVDW1QqJX27\nd6Fv9y54ebrTpX1bZkwai5eHO/Mfm8WMSWMdXXShmkQwqWVypZKB4x9EJpdjMZswGQ0ENA7DzcsH\nZzcPOvYbTsOmEWiLC/HyC8Rg0KEtKuTs0b04ubpz9vAewtp2pSA3i6DQ+tk+U5Fvv/+RCaOGM++R\n6dwzuD8vzJuN2WzG2UmDwWDA28ujdJ7WzOwcdHo9Umn9/bXctWtX6fb58+fJyak/gwIdTdzm1DK5\nXIF3gyAAOvcfSfL50zRtZZs2MSUplvwc2/M0hblZKFRqTAYDIS3bo1SpSb+YQGibzpw/eZChkx5l\n94+rCWrmuMbUmmjTIpyXF39MTl4e8YkXSc/MwtPDjWED+rLrr0OcPB1Ny/BQfvr9T1au+56P3niR\n33ftY9/Bo7zx/NOOLv51Bg0axNatWxk+fDhr1qxh4MCB9OvXz9HFqhdEMKkDao0z0X//RUFuJmaz\nicTok5QUFdCycx80zrZu5ayUJI7v+512vQZzfN9vAAye8DBxUUdxcnWn++CxhLXtisloJOHccUde\nTrXcf98oXF3KTj5lMpmQy+W0aRGOk0aNl6cH6ZlZDOjdDRdnZ1ydnQkNaeygEl9lsVhYsGABr7/+\nOkuWLOGhhx5i/PjxvP766/Tt29fRxat3RDCpAx363F2t9I2btyndbtdrUOm2t7+thuMfFGKfglWT\nRK7AYjJUnvAa/w4kQOmUkUGBV58+9ve9uni5t5cH3l7VX+XPajZhMWiRyGSVJ64CqVTKiBEj6NSp\nE4899hh+fn54eHgwefJkFixYgIeHfVYivF2IYFLLDHotOzd9hdGgRyKV4uHth1Su4NjubUhlMuQK\nJQa9jiYR7WgQ3BRv/yAuXThHx77D2LFpJYEhERzf9ytunr5cij+Li7snU+a+gbu3X51fS/PX9pCy\n8RVK4o8iVdl3fZ+bZ0UVEEarpbFI5fZ5xsdisaDT6YiKiuLQoUPk5dnmmhw9ejQ9evQgIiKCgQMH\n2uVctwMRTGqZUqUh43IiAY3DOHdsPy069sbTLxCNiysRHXtTlJ9DVupFlCoNSCTs+nEVTVt14ovX\n59C8XTckgJOLGyEt2hHWpgt/7/0Fva7EIdcikUgInPhqmX16vZ5t27Yxdmz1e18yMzNrPKG3yWQi\nMTGR0NDQah9bVVKplMGDBwPQrVs3AN577z0A/P39SUpKQqOxz/SVt4P622x+G1OpNTQJb0fXgfdy\n+UI0RoMeo942jF4qlWEy6DAZ9Di5eiCRSrlw5hi5mWns3rIaZ1d3Th7Y7uAruGrVqlV89NFHNZpH\n95NPPuGDDz6o0XnXrVtX+sWuSy4uVyddcnNzQ6FQ1HkZ6isRTOqAVCpF7eSMXK7AL6gJAGaziV/X\nLiO0TRfyslJp0qIdao0zXQeNwmqFcY+9QFpyPBaLhXGPLcBqMRPSogNdB46iQXDt/TWuDr1ej7e3\nN08++SSHDh2q1rE5OTn06NGDadOmcf78+WodazQayc7OplGjRiQnJ1frWKH2iGBSBxo0akZgSDiN\nw9vSILhvyFtnAAAgAElEQVQZYW274hcUQvP23SnIzeKhFz4hMyUZi8VCqy79GDLxERRKJVarheZt\nu2Ixm7lr1ANYrVYO7/wJpbp+VK3lcjnjxo2jadOmdOnSpVrHuru7M2zYMLp06UKzZs2qdaxOp+Pp\np59m6tSpNVqyQ6gdos2khqyAtIoLZg2e8AgAoa2vtg106D0UgBYdewHQf/QDZY4JDm3FlLmvA+Dh\n4w/A+McWVKuMUqmkVtdLll3TayKrZg/KzRzr6upaui16VOoPUTOpIU9XFTExFx1djAod/TsOD1cx\nDaJQN0TNpIbu7RfGvf3COB2Xic5gn0mbL126hIeHR5lGvprSqOXsWzm1yum/+uqr6/YdPny4dBXA\nypw5c6Za5btiwoQJNTpOqH9EMLlJrUPtt4DU5x++RJMmTViwoHq3M/bw4IMPXrcvJyeHyZMni3YJ\noUpEMKknEhISGDBgAAD5+fm4u7s7uESCUD0imNQTgYGBDBkyBC8vL/R6vaOLIwjVJoJJPaFSqSgu\nLi7dvlN06tSp2r051x57bc9OVSUmJjJixIganVO4MRFMBIdq1qwZzs41e84nPDy8RoE3JMQxD0re\n7kQwERzq008/xdfXfo3YguOIYPIvUqmU9u3blz7gVZdMJlOZmbzqG6M2g7jf78Wote8C6al2ycWK\nUZtFi1H7cPJuZ5ccheoRweRflEol27c75kG6+j4FYNR3oTQMG0CjiBlYrRZHF+c6BdkXOLulD+3v\nT0GmuPmxOnZhMWE16rCa9Pyy+QcupqTgUUlXe3pmFkrvIHx8fCpMl5mZidVqxc+v4ukoUlNTCQio\neOmQ/Px8CgsLCQoKqjBdUVERbm5ujB8//rr3RDARqkyu9iQ4fDAqTdlu6+OnkunQplHp68ij8eQV\naBk2oHWZbYCUtDw+W72XPt1CGdzv6py27y7/gwcn98LTw5nvNh9m8piu1S6f2qkTMUecsD3s4CBW\nC1aTHqtRDyY91mtWqD98/CQNfH1o3aJ5hVmciYnFy9mHNm3aVJjup59+wmAwVDqnyvHjxxkyZEiF\naSIjI0lMTGTYsGEVpouKimL37t0imAg3RypTl1sj+Xjln3z94QwAos5eIiU9H63ONiPbtdsAq78/\nSPdOIXy1/i98vV1p3zoYgC2/nyQuIYPPFk9j6/YoJo/pSuTReLp1DMFoNJOakY9eb1uG1WA0Exzo\niUwmxdWl7OMCtvJJWLt2LcOGDauztZutZiNWox6rvhBu8DyUk0ZDcMMAWoRV/GBjAz9f/IKCaNGi\n4vl+//77b/R6faXp/P39K02TkpJCSkpKpekKCws5duxYue+JYCLclEPHEjCZLOTll7B+yxHy8rWc\njrnM3f1b8dnqPew/HMfd/VuVpg9s4M5jz61l8Yvj2Pzr8dJgEhzoSWADD3buOwdAYZGO+KRMVm2M\n5PmnhjF59pe8vWAM817ayDOzBzPx0c/p2LYRa5aWHblborPy/gcfEx0TX+H0BM8///zNXbjVitWk\nK7cGcqcSD/oJN2XVxkgycwpZt/kweyLP88zswYwZ1gGAPZHnS7cBDAYTW347SdKRtzlx5iLHTpX9\nsv/3ybv55KtdaHVGjpxIZNq47uTm22aV6901lLt6htOyeQDTxnWnXasgLJbyawBXnpS2Wq3l/qvu\n/CnXsuiKsBRlY85PxVKci9VQIgLJP26bmsm5c+dQKpWlc2McPHiQ7t27O7hUtzet1kBKeh5bVz3J\nvdM/ZeKozjz14gYKi3QM7d+SoXe14sdfTzC0v61tRKGQoVDIWLhoCzl5xchkZf+WKZVynp8zjJEP\nLOW7ZQ+xcNEWurRvUvoe2KZVAK479gontYT5/5nL+o1bbnibs23btipfo9VqAZOhtBEVEThu6LYJ\nJl988QVWqxW1Ws3bb7/Ns88+y969e4mKisJqtXLs2DFmzpxp13Ne2DmF3KSf8WpY/TlMy2WF/KpN\nkVIpiURK9qXDBHd7B79Wj9slT7nam0uxf9I44m6sVgtSYMOyKVhMhfy40jYfy+R7yy4UduW1Xmub\njHnVB/eVef/K/q/fG49em0f7Fp5cPLIQrMW8+NTV9WhemnsXem0eK5eMK/3/2uPB1ptjMesACVOn\nVv2JaQCFXA4mHRadAatRW61jBZvbJpgATJs2jddee40TJ06U7nvxxRcBWL9+vV3PZTGVkJv0M33H\nL0MqrXg4eFxCBg383HBxtjUWFhXrSrfLew2g1xtRqWzzixqNZhSK6g85Lym8h+O737NbMAkdvJmo\n9WGkJRxCKrfPPClWqxVJFSeZqoxUpqHFqL+QKao2olZiMSGxGJFajAzr0wlLcW69Xk2wvrutggnY\nJimePn166X2zWq3GarUSGxtLu3Y3P5gpLy8PJycn5FIpEqmi3ECyc985Bva52iq+4aejjBvekYgw\n2zoxl1LySrdffW8rJpMFlUrOs48PRamUU1Ssw6/NM5z682WaNfFl+tyvWbfsoWqXVSqzz5IPVyic\n/Ok0q8CueWZmZtbtCFirFanFhEKfB9f0TNkroN3Jbrtg0qhRI8aPH8+HH34IwDfffFN6m2MP7u7u\nLFmyhKfmPFbu+5FH41n7v8MM7NOCXX/F8MO2YyQkZzF2eIfS10/MvKs0vUqp4NS5RGZO6sW8lzay\n7J0pAAzu24Ln39rMxs9tUz4WFGp5+5PfmDauGzm5xezcH03Txj4cO5WMTmekY9tGtIloSI/O1ZtP\ntbpy9q+n6MwuLMbqLcZ1IyaTkWL5zc/wbrWYUAeG02DcC9cFBonFhNRiRGIx2rbNujKBpK6cPBPN\n2k1beHfZygrTpaRnoJDL+fSDJRWmS8/KxmKx8u2331aY7tKlS/z4448VpsnNzaWgoIAdO3ZUmK6g\noOCGwf+2CSavv/566UNfjz/+OFOm2L6UTk5OAPTp06fKeT377LNs3Ljxhu9brVZ+2PQdyx+/vjfh\nq/UHiIlPI79Ay+dr9vLd8od5+uWNSJCUvo6OTQNs3Z9R5y7Rt3tzki5lk3z56gjYBn7u9OjUlK/X\n/wXA91v/ZmDvCD78YifDBrTC3U1Dvx7N2X84jufnDOOFd37k1LnLtRpMsnZ8QeqmN2g27TW7LXRl\nN1YrKbvXkrL2vzSctgip2WALHmYDEkcOYrtGu1YRTBl7D8MGVrw28WervsPb04P77q14ANnaH35C\nbzDwyPyFFab78MMPmTdvXoVpdu7cSWRkZGmzwI0cPnyYH374odz3bptgcu2TpxKJBE9PzxrnVVRU\nxHPPPXfDKQU///xzZjwwhdTtHcvsLykxcDElh4G9W7B+yxEMBjOFRTqSL9mCxJXXV8QlZNCwgQfT\nJ/Rg4mOf075VcJn8ZkzsyaTHvkBvMKFRK0nLzMfDzTaFYuMgWy+Fr7crbi5qfL1dKS6p3XlQ0ra8\nS8unvsQluH4unu4e3o1jr4yg4YSFyCz2qTkJVSdam27AxcUFb2/v6/5JJBJmzZqFv7//dcckXc7m\nuSfu5oW5w5BJpSyYO4zFn/7OiEFt8PRwKn3t6WGrLXVo04gWYQG8s/Q3Pnh1Av6+tmc2lAo5A3qF\nA/DOC2MY0q8l40Z0IDevhOeeuJvQJn6ENvHDzUVNz87NUKsV9OsRxsDeEbX6mchdvFA419/Z4KWK\nf6YjMJscW5A71G1TM6krV8YtWExarBZTmd6IFmEBtAizPVD10NTeAHRq27j0WH9ftzKvAWZN7lW6\nfeVYpVLOxFG2dWiaBPswe7qtWjznQdu0jj7eVx9iGzHI9vzG2OFla0lX2PsJX0G4ERFMakgq19Bx\nZh7nT7yDRCIHO/QGaEtK0PzTxmMPcpWn3XtfBOFGRDC5CRKJBC+/MejT4sEO4xMsxiKcLPZ6dF6C\nU4MOlSerJW9/vIIH7hvNa+8tZcrYkfj5eLFjXyQASoWCzOxcSkpKeP2/83jzw884cOQYk0aPQCqV\n8sCE0ZyLjSf5UgoKuZwBfXo47DqEqhPB5CbkHfmJiyvn4NGmv926Gu1Vj7AY9SSveJRmz2/FqXFb\nO+VaNXEJSezY8xcSiYTw0Ka888nnLFv0MufjE3jm8Qf5at0PvPJ/c3hp0UfMmreAAH9f3nphPivX\nfk/HtraHAmMvJHLk+CkG9e1Zp2UXak4Ekxqy6Eu4uHIOred/i1ODpo4uTrmyT+7k8upnCVv4W52e\nNzSkMT27dOToiVMUl2hxcXbixOlo9hw4QmADf0q0Wv7YvZ/4pIvMnjGZDT/+woSHnuLh+ycyoHd3\ntFod7q6utAwPRamsZ13QNyE9M4v0zCyycvIqTHfo2ElcnJ3R6SvukYo8egyjyYx69eoK0x05coTV\nlaQ5c+YMcXFxlaaLj48nLS2t3PdEMKkpiRSpUlNvAwmAa5O2mIuW1vl5T5w+x+4Dh3hwyn18v/U3\n5HI5jYMb0rNLBywWCxNHjyAnN49uHdux/9Df5OTlM3PyONxdXZj48FwWv/Qs+w8f4/dde+naoS07\n90XSr2cX+nSz0zNQDmSxWDCbK35Y0Gq1VimdxWJLZ8iqeJlaU3FeFfKyYLVaq5yuPCKY1JQErDfZ\nBVlYVMwT/30VpVJB766d8PPxRiqVIJVKOXEmmoemjMfL8ya6Ymtx0fKKtG/dgkF9e9KmZThr/7eV\nDm1aEnUmGhdnJzIyszl64hRmi4XkS5dpFRHGxFHzePy5VwhvFsJv61fi5elB907tOR19nuISLTMm\njaNNJbOT3Qr8fX1oFR5a6aA1nV5fpUFrCoUcvcHAjEljK0yXX1hU6UOuVwatVZbujhi0dqtJy8hk\n3sK3aNsynLt6dmPF6vVMnzDa9iW7nEpi8iUUipsfZu4I8YnJSCRSOrVtxfQJozl3Pp6xwwdjNJlQ\nKhRMnziGH3/dwdmYOM7GxHEpJZ03nn+azu1sUzsmX07h/15dzIbPP6SouJhFn3yBTCalZfNQB1+Z\nUBExaM1BsnPzaBUexvFTZzl++izNmzXhjQ+Wc+hYFKnpmew7eJQPP1/FGx8sJy4hydHFrZZmTRrh\n4uzE82+8R+LFFAxGI82aNCL5UgoAryz5mDc/XE5gAz9CQxqza/9B/tx/kHumPcrLiz/G3dWVl+Y/\nwcJ3PiTpUgqTxoygedMmjr0ooVIimDhIRGhTvDzdWfifJ9j002+YzWZ+37CSx2dMRq/X07l9awoK\nC3lwynhCQxpXnmE9cyHpIn17dKZT25akZWQReyEJlUrFlLH38PxTj9Gzcweemf0gp6Njkcll/Lkv\nEp1eT+LFy5w9H8eR46fIzS/gyPFTvPD2B+QVFDr6koRKiNscB8nIyiE6Nh6DwcgX779RGjCefult\nnn50Bu1bt+BSShpvfbScd158Bmc7DmarC3KZjAZ+tqdLNWo1zZs1YWCfHhw8dpIdew5w+HgUb330\nGZ7ubvTt3qX0uMupafTo3IEenTvQrVM7WoQ141xsPBr1nbNk6q1KBBMHCfD35ZO3Xrpu/6pPFgG2\nVvOgwAblprkVFBYX88vOPQBcSrV1JX65diMrP3iLPt06U1RczKvPPsUbHyynpKSE5Mup9OjSgcKi\nIgCMRiMRobaeMp1Oj5NG45gLEapMBJM6ciYmloN/n6SouIR7hw7AYrGw/sdtvDBvNlt//xOAkUMH\nsPSrNVitVn7evpt7Bt+Fk0bDg1Nsa5Rs3PILE0YNd+RlVMu1nUkZWdl4ebiz+NMvKS4uIfLoCV5e\n/DE+3p7k5efTonkz4hOSiQhryqoNm9m2YzcL5j7GH3v+4sz5ODF50S1ABJM68tZHnzFp1Aj8vL14\nd9lKnnroAYbc1ZulX60hJu4CcQnJHD15GoCoczGMGjqQNZu2MHH0CPZEHmbb9t2cjo7l6MnTmExm\n3n/tJpdqqGVzH55O25a2J5+jzsbg7KTh6UdnEtjAtvrcq88+BdiWRC0qLuF0TCyNGgbQqGEger2B\ngX164OHuipNGw9OPTHfYddhTdSdH+mzVdxWmuzI50tpNP1WY7lJqOj/9VvGkR7n5BRQUFonJkW4F\nuXkFRJ2LAWzjCGQyKX/uP8iDU8ZzJjqWvQePMmPiGL5Ys5GeXTpQrC1hzIjBqFVK+vXoSr8eXVnw\n5nu89cJ8nnz+NQdfTeWuBJJrt8tr95HL5Xi4u9G7a6fSfSqVkqBA27SWzZtVbT7XW0FtTY40a/L1\nq+td66MvVjP34QcqTPPnvkgi/7bVFitS0TgT0ZtTR5o3bcIj90/kkfsn4uXhjlKpQKvV4unuRgM/\nXzw93Ahs4IeTkwalQsn5+ET8vL1JTL5MiVbLx1+uxs/Xh7z8AgqLih19OYJwHVEzqSN5BYWsWL0B\nALlMTkxcAv17defND5eXpnnzw+XMefB+Vm38kdCQxhQUFfHo9Els+W0nGrWah6dNYPazLzNr8jhH\nXYYg3JAIJjUmqda0A998/E65+/v1vH6B7nn/aiNoEtywdHv54lerfE4kEpDYsfJptSCpZFkPh5PK\nbC2/or22zonbnJqymJEq1OhzUhxdkhsqvhSNMa/8JzxrQu7hT150pN3ys7fiSzFYdEVI7BlAhSoT\nNZMakqqcCBi/kJNv34dPt1EOWTqhIlaLheyj2wh52n6LjwXdv4TY14eQe+4v5M41n7C7VlgtZB/9\nhUaPrgCVMxgKoZ7MSn+nEMHkJnj2vA+lXxP0qbFlpm08ceIE7du3r3Z+ZrPlhmvoVoXVem0xJIQN\nm4u6YXhFh1SLumE4EYuPUnRuP1bT1bk2kpKSaNy4ZkP+zWYzMln1b50KCwtwdXW7ukMiIXToHDTB\nLbECRrVn6Wp9UrMRrGKN4NomgslNcg7tgnPo1eHgVquVl59fxr4nF1U7rzVr1tCsWRN69Kj+NIW/\n//47Wq2W0aNHV/vY6lC4+eLZbUzp67S0NBZ9Mo/165+tdl4HDx4kJi6G6dOnVPvYj15+mREjRtC1\n6/VtTldYpQrMUgVmOWA1IzXbFuJyVINKbn4Bx0+dQy6v+Gt3LjYeNxcXPNzdKkx3+tx5jCYT2/f8\nVWG6mLiEStOcOHWWuIRk/vjjjwrTRUdHk5OTU+57IpjY2ZkzZxgxYgTHjh2jY8fyZ4wvT1xcHH37\n9kWhUJCRkYGfn1+Vj01NTaVt27ao1WoSEhIICQmpSdGrzWg0YrFYmDZtGrGxsYSFhVX52MzMTBo1\nakRAQEC1j71w4QKTJk0iMDCQ3Nzcqq2RJJFhkcsANSaFM0a1bZUBidWC1GIkLSuXdrW8zrCnuxut\nwkMZ3K9XheniE5Px9vSoNF1GVjZ6g6HSdGfPx1eaRiaVYjCZGNA6uMJ0LsZcLkZHlfueaKmys9at\nbXNyVCeQAISGhuLi4kJAQEC1AglAQEAAAQEBuLi41FkgAVAoFAQGBgJUKxgA+Pr6EhgYiLOzc7WP\nbdq0KTKZDHd395tabA3AKpFilqk4FBWDxNUPqYs3SMXf2JoQn5oglJIgkauQuflhtZjAZMBq1GE1\n6hGNuZUTwaQW/PDDD5w+fbraxxkMhpuaQPnrr7+u8bE3Y8GCBSxaVP02IrA9m1NZG0J5dDodR44c\nqdE5q0IilYNSjkRpewTAatJjNeqx6otq7Zy3OhFMakFliz/fSFFRES4uNV83pyZfSnuobEbziuTl\n5eHhUX+XHL1CIlchkatA7WwLKkYdVpPeYfPs1kcimNSCUaNG1ei4nJyc0uVHbyU16Qa/IisrCx8f\nHzuWppZJZEiUTldrLEb9P7UWHVju7DWORTARhJsgUaiQKFSgccNqMWPV5tvG4NSzQYx1QQQTQbAT\niVSGxNnLdutjNpbWWKxmo6OLVifu+GCiUqkwGo2o1eoy+0+fPs1vv9XtSnhFRUWMH1/x3BQ3y2Aw\noFJVbT5VhUKByVS7VffDhw/j7u5eq+f4t+joaEaMGFH62mw2I5VKazQSt1wSCciVSORKJGrX0t3P\nvPKObRGrSmaNe/g/LyCRSCpNN/nRpwEqTffkf1/BWskYmrvGNKPf6KmVpus8uBldh5b/1PodH0y8\nvLzIzMzE1fXqD71169bs2rWLCxcu1GlZDAZDrQaTgoICFApFlYOJl5cXWVlZtdqO89577zlkfaD5\n8+eXbmdlZdVJI7CjGsjrisR6o7X+7hAHDx4kJiaGsWMrXhWtLtR2A+wff/yBUqlk5MiRVUr/999/\nc/LkSe67775aK1N9aIDdunUrfn5+DBo0yKHluNXd8SNgO3XqREFBAf/73/8cXZRad+LECfr1q3jK\nwGu1a9cOrVZbiyVyvO3bt3Pu3Dl69uzp6KLc8u74mgmAVqvlrbfeIjg4mGHDhjls3ENt1UxKSko4\nduwYPXv2rHb7hE6n45tvvmHYsGG1UjZH1UwKCwv5+++/OXPmDHPmzMHZ+faZa9ZRRDD5h8FgYPfu\n3Rw7dgypVIpGo7nhau+1pbi42K6/1BKJBJ1Oh16vp2XLljW+ldu+fTvHjx/HYrGgVqvt+rnc7EC9\nmpBIJJSUlNC2bVv69etXpr1MqDkRTMqRn5/vkOp9lZ+ArQaVSoWbm5tdeiqKiors/rk4omYikUjw\n8vJCWstPCd9pRDCpR27VEbA3IzMz84brsAi3FhGaBUGwCxFMBEGwi9t7FE0dsFqtnD9/nv379xMb\nG0t2djadO3emR48etG7dWtyXC3eMG7aZpOfoWL4lkZxCw527aLTVSrswd2YNK3+y5Pz8fBYtWkRi\nYiL9+/enQ4cOeHt7s3//fnbu3Im3tzcvvfRSlbtjRZtJ5axWK+fOnSMyMpIzZ87g7e1N69at6d27\nN97e3jc8bv+pbE7E5mM033kP4AFYLODjrmDa4EbIZLXzfS43mBTrTIx+4RC9WzegWaAblju0idZs\nsbDnZAqdwt2ZM7Zpmfd0Oh1Tpkxh+PDhzJw587reEqvVyqeffsp///tfEhMTq9RjIYJJxdLS0nj3\n3XfJzMykX79+dOzYkezsbA4fPsyhQ4cYOHAgTz755HV//NbtuMjXvyYztHMQUumd+YdRJpUQeTad\n7AI9W9/ujlpp/8XUyg0mG3ddZn9UHvf1a2b3E95qDEYzH20+xQ+vdUUhv3rLMmTIECZPnszIkSNR\nKBS4ubmVW4P77LPP2LdvHytWrKh0PIUIJjeWnZ3NzJkzmTlzJqNHj77usy4oKGDRokXExcWxdu3a\n0udgtHoz9y44yLxxbXBzqvksdreL9btiGdjJm7F9A+2ed7k39EVaI/6e169Yb09HIndiNps5e+ow\nACXFhZyJOlTm3+WLVx+0i489TXraxTJ5mIxGUi4nUpBf/tT79qBUyLBYrBhMV6vHVquVQ4cO0a9f\nP3x8fHB3d2fdunVIJBLi4uLKHP/YY4/xySefMHXq1Bs+gfvtt9/y+eefI5FISE1NZdiwYfz555+1\ndk31walTp/jiiy9Yu3YtBw4cqDDtkiVLePrpp5HL5axcuZIRI0bw6qtll0l1c3PjzTff5LvvvmPW\nrFml+01mKxqVrFYDSVZmKnExUej1utJ98edPXff7fPb0EfQ6re3356/rl5RIuZxIWkpSrZUTIMDL\nmcKS2nkS3CENsAaDnh/WLyM5MYaUy4mcO32UpqGt+Xnz12RmXMbdwxtXN080Ti706T+SP//YhK9v\nIGaLmbzcLLr2GMR3qz9kyPBJAGRnpTHjkQV1Vv4zZ84QHR1NQEBA6b6pU6fi5OTEsmXLeP/998uk\n9/LyonHjxpw5c4Z27dqVeS86OpqTJ0/y7rvvlu7bsmULH3/8Ma1bt672TPW3giNHjrBu3TrmzJmD\nRqNh/vz5FBQUcPfdd1+XNi8vj9WrV6NUKtm6dSuBgYFotVoeeOABtFotGo2mTHqpVMqjjz5aV5cC\nwKZ1n+Lp5cvJY/sBaNOhJ9+uXIxBryMgsDEX4s4glckICGxMSGgrTCYjRYX5HI7cQffeQ9m0dinB\nTZoTFt6W/Xu28fqSdXVafntxSDDZ8esGGgY3Q6V24oGHnuO3rWswmgyERbSjaVgrYmOiCA1vS0lx\nIb/+9C06bQlBwbZbLoNBj69/Q5qHt2PAkPHk52Vz/OjeOi3/0aNHmTFjxnX77733XlasWFHuMe7u\n7ixfvpzmzZuX2Z+UlMTYsWOJjCy7hm+LFi1ISEi47YJJVlYWy5cvZ+nSpTg52Wq/a9eu5ZNPPiEk\nJITw8LIrEG7evJmCggLi4uJKpyrQaDR8//33fPnllzz00EPXnaNbt27odLrr5qipDVptMdt/Wc97\ny7aSlZlKQvxZXFzcaR7eDmdXdw799Qd3DR5L8T/BIy01mS49BuHh4U1uTiY+vgFIZTImTJuDn38Q\nSYnna73MtcUhwSQvNxOVWoPJZCQ97SJOzq507TGITeuW4uzijtls4kjkDoaPmk7T0FZs3riCqOMH\nkMnkOLu4sWndpzi7uPHD+mWUFBdxKTmOEaOn11n5//0Lf4VMJiv9gvzbgQMH0Ov1172fnZ0N2O75\nryWRSLBYbr+eh8TEREaPHs3582W/NO3atWPVqlXXraGzZs0a+vTpU+6cJ0ePHi03mMjlctLT02nY\nsKF9C1+O6DN/07hpOP6Bjdi6+WuKCvO4b+qTKFVqTp88iFKpKv0/uHEonl7+pKckk5+Xjdli5u9D\nu8lMv8zXn72Bf0AjLiae5+Sx/bTr2LvWy25vDgkmU2bMZ+Wy1+h91z1s3rCC+6Y+CYBKpaFJ0wgA\n8nKzMJtNKJUqLGYzRUX5yGVyJFIp4yY9RmZGCn/t3kZwkzDc3Ot2Ee2KGgz/XfMA23IOycnJfPTR\nRwwfPrzMe6dPn+bw4cNl7vOtVivz58/nmWeesV+h64krS1skJZVtG/Dw8CA6OprU1NQy+w0GA337\n9i03r5KSkhueJyEhoU6CSYfOfTl2eDcajTPakiIah9j+0BgMeho1aV7aUJyeepFmzdtwJuoQCrmC\n4uICzGYTXj7+9Oo3AqlMhtVqpWFwM4wGQ0WnrLccNmitID+HP3//HqlMxh/b1hHRqjN5edmkpyYD\nUG6mUP8AAA9HSURBVFiYT3JiDB6efnTtOZic7AyUKjUGvQ4nZzf+2v0FoeFtaRDYiOQ6rhqGhoaS\nmppaps3kinnz5rFr1y769+8P2J4EbtiwIYGBgQwZMuS69C1btuTAgQM8++yzvPHGGxiNRt5++216\n9OhRulre7SQ4OJijR4/yxBNPlOmRef/991m4cCEdOnQok/7nn38mOjq63LwyMjLKXQ41LS2t3KBe\nm/78YxM+foFkZ6axf9dW0lKSsFjMpdeYm5NJRtol3Nw8CQwKIaCkiNTLiTg5uZCWkoRer2Py9Kf5\n3/rl+PrXfhCsDeV2DX/1SyKpWVb6tr3+y2IvMeeO0zDo6tiN/LxsdLrr/9I0bhJOUWEeMoUCmUxO\nWkoSDYObYTDokEllJCXEENSoGa5utVc7WbT+OBte6YKz+mrsfeutt1iwoPxG30mTJqFSqXB3d+fE\niRN4e3tz1113MXfu3Bue4/vvv+f8+fPI5XI6dep0W8/6tWnTJnbv3l06ZeM333xDQkICr/5/e/ce\nFWW5L3D8y224yFVkVEQFhRBBdpqQB1G2d5flqnboSttesmPHZeHuaJq2RNlKrDBb7pVnYakLLRaE\n5TFR0UrJc1JTRCXoDFiD5OaSiFAMzDgMw8x7/uDIycRLwwsvyvP5x7XkfZ/5/dYafjzv8z6Xv//9\njhnDtbW1LFu2rMPNq06fPs2ECROora1t7y3W19cTEBDQvrq56WYrLyZfYPVc24/juB/tD0UEDAnG\n8n9v61pamvn1lxt3XHfru2wwNDFwUCCXNRcIi4jil/rruLi4oW9qwGRqZkhg1xXC/y76mcH9HVk0\nc4jsbXfYM5EkunxyT2jY7X+B3D3uPkvUu+//P1YMCw4HwNm5bXAtLGJsF0R3Owd7uztOh9RoNGRk\nZLBgwYI7rs/MzKSqqgqj0Uh1dTUrVqxg9+7d9/yMOXPmUF9fj5eX1yO/V2h8fDx2dna88847ODs7\nExQUxIYNGzpceqBWqwkMDCQ3N/e2TaABYmNj2b9/Py+//DKPPdb2SJGXl9dhD7ArhYT+6Xf/40Vf\n3/4dXuvd16/9+zxyVDQAvv0GANDH3bPLYuwO95wBO6hfH6JD/XrtDNhWi5Xc/AoWTB/Mwhl3VnKd\nTsfatWuxt7dn6tSpPP744+3T6U+cOEFDQwMbN25k6NCOp+P/Vn19PVu2bMHJyYmkpKRHvqAAfPrp\np5SUlJCUlHTfa8+cOUNsbCxFRUVERkZ2eE1GRgYZGRkcOnTotjc572Zr0VYaifuTf9sfhl7I3s4O\nbbWO/yr6mc+Tn8TdVf7v113X5phbraQf/Sd1ut67NkflaMeYx7yZPObuA64Wi4Vz585x6tQpysrK\n2hf6xcTEMGHChAcuCrW1tTg6OuLl5UVDQ8M915k8CiRJYs2aNRgMBlJSUh5oq8zvvvuO5ORknnzy\nyfa1UHV1dVy4cIEjR45gb29PUlLSHQPkkiTxj/1XKP2nnl5aS5Ak6Oel4rW/DGOgb9e8MhebI/Ug\nvWk6fV1dHR4eHuh0Osxm8wO/edHr9Rw5coRz585RUlJC3759CQ8PZ/LkyYwbN06+s2+EP0wUkx6k\nNxWTW8ROa4+OR//BvIuZG+swXiu7/4UP4GZjI4435BmEs7N3wG1oBA4q1/tf/AdYjE0YKkuQLPKs\n77jZ0ICuTobTACQJ10GhqLxEYVKK6Jl0gv6nIorXT8IjJLrtSMgexNxYR3PNFZ54/3ucfeWZt9Dy\naw0XXhuJ+7DR2Ku6fqr6H2FtbUVfVkDk5jzch42+/w2C7EQxsZG1pZlzL/kT+u8Z+I596v43KKDy\nP1Np+P4ko5LkOTP52xf74j/rVQJf3CxLe3JrKP4a7YevMWbbRdl7ZML9icecTrBzVNlcSKqqqsjK\nyiI4OJji4mKgbfwgMDDwtnU6ISEhLFy40KbPUP/5r1w/+bFN93bEuV8AA6Yvtene7sjXO3IyFmMj\nWC023S90jigmCgkICCAyMpKrV6+2L2IzGAysXr1a4ci6Rm/LtzcSxUQheXl5HD9+nMWLF3Pp0iV8\nfX1pbm4mMTGR/Px8wsLCqK6uJiwsjMWLFzN8+MO9611vy7c3ElunK2TKlCm4uLhw7Ngxxo0bx5kz\nZ7Czs2PDhg2Eh4eTkJDApEmT2LRp0yPxi9Xb8u2NRM9EQZWVlVRUVGA0GvH09MRoNJKWlsb48eOB\nttm1hw8fRqVSdbgL2cOmt+Xb24ieiUJaW1tRq9WkpaWxceNG+vXrh729PW5ubsTHxzN8+PD2Yx0m\nT56sdLid1tvy7Y3Eq2EbWVuayV8ayL98VNMl7Vsslk5PDTfVV/P9xumM/Q+NLDFd/Fsk4euP4OIn\n//J1OfIFOLckgKi0yzi43PskAEF+4jGnm6xbtw5JknB3d2/fRBnatjLYtm1b+xYFmzZtAuCrr75i\n+vTpeHh4sGLFioduzcmtfGtqahg1atQjn68gikm3+PbbbyksLMTT0xOTyYS/vz+lpaWEhYWRm5vL\nzZs3KSwsZPTo0ahUbUcyODg4oFKpMJvNZGdn88MPP9zW5ty5c4mIiFAinfv6bb5ms5mysjKb8j1/\n/jzR0W17fvTkfIU2YsykG8TExODh4YGbmxszZsxg4sSJhIaG0tTURGlpKenp6WzduhVJkpg2bRon\nTpwgLCyMkJAQampqmDNnDmazmYSEBGJiYoiIiOjRv1i/zdfDw8PmfENDQx+KfIU2oph0k8GDBxMS\nEsITTzxBbm4ujY2NLFiwgNWrV+Pi4sILL7zA8uXLSUlJITIykiVLlrB9+3bGjBlDQUEB3t7e+Pn5\nMXPmTL7++mvKy8vv/6EKupUvYHO+JpPpoclXEMWk2zg7OxMcHMzHH3/MwoUL2bdvH2+//TYjR44E\nYPbs2cyYMYP9+/dz5coVLl68yLFjx7BarTg4OLBq1SoSEhKAts2XhwyRfxBUTrfy3bZtm835xsXF\nAQ9HvoJ4m2Mza0sz5/9tOOP2VCsdyl21/HqN4g3TGLv9f2Rp7+LfIonYcBRn3wBZ2usK+f8ayNjt\n3+Pg6qF0KL2O6JnYSJKs2Ktcab5RoXQod6W/UkjLLz/L1p6jhy/68u9ka09uzTcq2xb62YmvtRJE\nz6STdJpTNJUVAJ3fz8RsNnd4cp0t7Bwc8AqfiHuQvEc8GK+V8WvhV0gWsyzttbS0tL/R6QwJCfeg\nx/GO+HPngxJsIopJD7Jt2zaCg4OZPXu20qF0i9zcXDQaDWvWrFE6FEEGoj/YQxiNRoqKijh+/LjS\noXSb48ePo9FoMBgMSociyED0THqIq1ev4ubmhqurK0ajEbVarXRIXerGjRu4uLhgMBgwmUwPdLaQ\n0LOJYtKDiN3phYeZeMwRBEEWopgIgiALUUwEQZCFKCaCIMhCFBNBEGQhiokgCLIQxaQHcXfvfVsN\nenrKc7ayoDwxz0QQBFmInokgCLIQxUQQBFmIYqKAiooKZs+ezalTp7hw4QJTp05lxowZ7Qd679+/\nn5UrVyocpbzee+89oO184ZdeeomsrCwA9Ho9AFu2bMFisWA2mzGZTIrFKdhOFBMF6PV6TCYTubm5\nxMfHM2zYMNRqNfHx8QDEx8ej0Wi4ePGiwpHKY9euXRw4cICkpCSmTZvGrFmzyMjIYOvWrbzyyisA\nFBYWYrVa+eabb/joo48UjliwhTjqQgFVVVX4+voyc+ZM9Ho9arWap556irNnzwJQU1OD1WolOTmZ\nzz//XOFoO8/Hx4eRI0cyYMAApk+fTklJCbNmzaK0tJSmpia0Wm37v1VVVUqHK9hIFBMFODg4oNPp\nOHr0KKWlpVRUVNDS0tLevd+xYweJiYkUFhaSlZXF/PnzFY64c/Ly8tBqtQwfPpzKykqcnJxwc3Oj\nT58+xMXFkZOTQ3V1NYcPH6axsZEJEyYoHbJgA/GYowBfX1/8/PzQ6/X8/s38oUOHKC4uZuLEiSxf\nvpydO3ei1WoVilQeKSkphIeHExQUBIAkSeh0OgDc3Nx44403GDFiBCtXrmTdunWcPHlSyXAFG4li\nogC1Ws2YMWNYunQp0dHRBAcHM2/ePBoaGkhLSyMzMxMAJycndu/eTXx8PD/++KPCUdsuOzub/Px8\niouLMZvNSJLUfuTFrQHYW6xWKw0NDUqEKXSSmLSmkOLiYvz8/MjJySEsLAyz2YxKpSIqKgpXV9fb\nrr1+/TpqtRo7u85vWq2UtWvXMmfOHPbu3UtQUBAWiwVoO3vY2dkZo9GIi4sLra2t+Pn5kZqaqnDE\nwh8mCYKMCgoKJKPRKKWnp0uSJEmZmZlSbW1t+8+rqqqk1NRUqa6uTtq6davU3Nx82/3r16+Xmpqa\nJEmSpOrqaunQoUPSrl27pGvXrklz586VTp8+LZ0+fVoqLS3tvqSEByIGYAXZ7Nu3D4PBwNixYzl/\n/jxDhw5l+/bt5OXlMXjwYJKSkkhOTub5559HpVJRUlLCu+++y9SpU9mzZw86nY7CwkJqa2sBeO65\n58jJyWHSpEm8+uqrDB06lKKiInbt2kVGRobC2Qq/J4qJIJvMzExycnLQaDRMmjSJuro6pkyZQkJC\nAv379+fq1avs3r2b8ePHk5iYyMmTJ3FwcGDQoEHs2LGDp59+mmPHjjFs2DAAvvzyS0pLSzEYDCxZ\nsoT09HRKSkr45JNPGDFiBEaj8Y5HQkE5opgIsjCbzTg7O2NnZ4ePjw/Z2dk888wzZGVlodPpcHR0\nxMnJidjYWKKionj22WcpLy9n586dAGzatAmNRnPbWMkHH3zAqlWrCAoKYt++fXzxxRckJiai0Wjo\n168flZWVjB49WqmUhd8RxUSQnb+/Pz4+PixatIgDBw5QW1vL+++/j4+PD2+++Sa+vr64u7ujVqux\nWCycPXuWgoIC4uLi+PDDD9vbOXjwII6Ojri7u+Pq6sqyZcuIjo7mrbfeIiYm5o7X6oKyRDERZOHk\n5ITJZEKSJMxmM+Xl5WRlZXHw4EG0Wi3r1q0jPT39tnsWL15Meno63t7eZGZmMm/ePPLz89t/rlKp\nWLJkCVFRUVy6dAmtVktqaiopKSkMHDgQLy+v7k5TuAdRTATZzJ8/n7179+Lo6Mjrr79OUVERK1as\noLW1FScnpzvmysTGxrJ582Y+++wzPD09aW5u5vLly+0/X7RoEWlpaZjNZs6cOcOQIUPYs2cP/v7+\nlJSUtM9VEXoGMc9EEARZiBmwgiDIQhQTQRBk4fjTTz8pHYMgCA+5Rp2O/wUsqgYgG2sJfwAAAABJ\nRU5ErkJggg==\n"
    }
   },
   "cell_type": "markdown",
   "id": "00edde14-0c43-47ac-9d9a-514e15813fc3",
   "metadata": {},
   "source": [
    "# transformer\n",
    "\n",
    "![transformer结构图.png](attachment:ad44b9b3-fffd-465e-813e-1fa785bafb31.png)\n",
    "\n",
    "基于Transformer 的编码器和解码器结构如图2.1 所示\n",
    "左侧和右侧分别对应着编码器（Encoder）和解码器（Decoder）结构，它们均由若干个基本的Transformer 块（Block）组成（对应图中的灰色框）\n",
    "每个Transformer 块都接收一个向量序列作为输入X ，并输出一个等长的向量序列作为输出Y\n",
    "Yi 是当前Transformer 块对输入 Xi 进一步整合其上下文语义后对应的输出。\n",
    "\n",
    "# 几个transformer中的机制概念\n",
    "\n",
    "注意力层：使用多头注意力（Multi-Head Attention）机制整合上下文语义，它使得序列中任意两个单词之间的依赖关系可以直接被建模而不基于传统的循环结构，从而更好地解决文本的长程依赖问题。\n",
    "\n",
    "位置感知前馈层（Position-wise FFN）：通过全连接层对输入文本序列中的每个单词表示进行更复杂的变换。\n",
    "\n",
    "残差连接：对应图中的Add 部分。它是一条分别作用在上述两个子层中的直连通路，被用于连接两个子层的输入与输出，使信息流动更高效，有利于模型的优化。\n",
    "\n",
    "层归一化：对应图中的Norm 部分。它作用于上述两个子层的输出表示序列，对表示序列进行层归一化操作，同样起到稳定优化的作用。\n",
    "\n",
    "# 嵌入表示层\n",
    "\n",
    "对于输入文本序列，先通过输入嵌入层（Input Embedding）将每个单词转换为其相对应的向量表示。通常，直接对每个单词创建一个向量表示。\n",
    "由于Transformer结构不再使用基于循环的方式建模文本输入，序列中不再有任何信息能够提示模型单词之间的相对位置关系。在送入编码器端建模其上下文语义之前，一个非常重要的操作是在词嵌入中加入位置编码（Positional Encoding）\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fa6bac9-591a-431d-9eff-ec4a4e0c2fd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "class PositionEncoder(nn.Module):\n",
    "    def __init__(self,d_model,max_seq_len = 80):\n",
    "        super().__init__()\n",
    "        self.model = d_model\n",
    "\n",
    "        # 根据pos和i创建一个常量PE矩阵\n",
    "        pe = torch.zeros(max_seq_len, d_model)\n",
    "        for pos in range(max_seq_len):\n",
    "            for i in range(0,d_model,2):\n",
    "                pe[pos,i] = math.sin(pos / (10000 ** (i/d_model)))\n",
    "\n",
    "        pe = pe.unsqueeze(0)\n",
    "        self.register_buffer('pe',pe)\n",
    "\n",
    "    def forward(self，x):\n",
    "        # 使得单词嵌入表示相对大一些\n",
    "        x = x*math.sqrt(self.d model)\n",
    "        # 增加位置常量到单词嵌入表示中\n",
    "        seg_len=x.size(1)\n",
    "        x = x + Variable(self.pe[:,:seq_len], requires_grad=False)\n",
    "        return x"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
